Reject zero-level Index results from string expansion - #24078

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Identify every accessor path that delegates expanded results to BaseAccessor.
rg -n -C 6 --glob '*.py' '\._return_or_inplace\(' python/cudf/cudf/core/accessors

# Inspect explicit expansion calls and add empty-Index parity cases for each caller.
rg -n -C 8 --glob '*.py' 'expand\s*=\s*True|expand=True' python/cudf/cudf/core/accessors

Repository: NVIDIA/cudf

Length of output: 50368

🤖 get_repo_knowledge executed:

get_repo_knowledge NVIDIA/cudf /tmp/coderabbit-repo-knowledge/nvidia-cudf-5d380a13/conventions /tmp/coderabbit-repo-knowledge/nvidia-cudf-5d380a13/learnings

Length of output: 8614

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- base accessor diff and implementation ---'
git diff -- python/cudf/cudf/core/accessors/base_accessor.py
cat -n python/cudf/cudf/core/accessors/base_accessor.py | sed -n '1,150p'

printf '%s\n' '--- all explicit expanded calls ---'
rg -n -C 12 --glob '*.py' '_return_or_inplace\(' python/cudf/cudf | \
  rg -B 12 -A 12 'expand\s*=\s*True|expand\s*=\s*expand|expand=True'

printf '%s\n' '--- expand definitions and callers ---'
rg -n -C 8 --glob '*.py' 'expand\s*:\s*|expand\s*=\s*True|expand\s*=\s*expand' python/cudf/cudf/core/accessors

Repository: NVIDIA/cudf

Length of output: 50367

🏁 Script executed:

#!/bin/bash
set -euo pipefail
git diff -- python/cudf/cudf/core/accessors/base_accessor.py
cat -n python/cudf/cudf/core/accessors/base_accessor.py | sed -n '1,150p'
printf '%s\n' '--- expanded calls ---'
rg -n -C 10 --glob '*.py' 'expand\s*=\s*(True|expand)|expand=True' python/cudf/cudf/core/accessors

Repository: NVIDIA/cudf

Length of output: 28553

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- definitions ---'
rg -n -C 20 'def _return_or_inplace|if expand|Cannot infer number of levels' python/cudf/cudf/core/accessors/base_accessor.py
printf '%s\n' '--- call sites containing expand ---'
rg -n -U -P -C 3 --glob '*.py' '_return_or_inplace\([\s\S]{0,500}?expand\s*=\s*(?:True|expand)' python/cudf/cudf
printf '%s\n' '--- extract implementation ---'
cat -n python/cudf/cudf/core/accessors/string.py | sed -n '570,670p'

Repository: NVIDIA/cudf

Length of output: 11671

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- every expanded call in accessor sources ---'
rg -n -U -P --glob '*.py' '_return_or_inplace\([^)]{0,1200}expand\s*=\s*(?:True|expand)' python/cudf/cudf/core/accessors
printf '%s\n' '--- string methods around partition and extract ---'
rg -n 'def (partition|rpartition|extract)|expand\s*=' python/cudf/cudf/core/accessors/string.py
cat -n python/cudf/cudf/core/accessors/string.py | sed -n '480,680p'
printf '%s\n' '--- current change summary ---'
git diff --stat -- python/cudf/cudf/core/accessors/base_accessor.py

Repository: NVIDIA/cudf

Length of output: 10784

🏁 Script executed:

#!/bin/bash
set -euo pipefail
python3 - <<'PY'
try:
    import pandas as pd
except Exception as exc:
    print(f"pandas unavailable: {type(exc).__name__}: {exc}")
else:
    idx = pd.Index([], dtype="object")
    cases = [
        ("partition", lambda: idx.str.partition("-")),
        ("rpartition", lambda: idx.str.rpartition("-")),
        ("extract_one", lambda: idx.str.extract(r"(a)", expand=True)),
        ("extract_two", lambda: idx.str.extract(r"(a)(b)", expand=True)),
    ]
    for name, fn in cases:
        try:
            value = fn()
            print(name, type(value).__name__, getattr(value, "shape", None),
                  list(getattr(value, "columns", [])))
        except Exception as exc:
            print(name, "ERROR", type(exc).__name__, str(exc))
PY

Repository: NVIDIA/cudf

Length of output: 215

🌐 Web query:

site:github.com/rapidsai/cudf "Cannot infer number of levels from empty list" base_accessor.py

💡 Result:

The error message "Cannot infer number of levels from empty list" is typically associated with the cuDF library, specifically occurring during operations involving MultiIndex handling or column accessor initialization when data structures are empty or improperly formatted [1][2]. While there is no single, globally defined exception in the public pandas or cuDF API with this exact string, it points to a validation failure in how cuDF processes hierarchical data (like MultiIndex levels or nested list columns) [1][2]. Common causes and contexts for this issue include: 1. MultiIndex Construction: It often appears when trying to create or process a MultiIndex where the levels or codes are passed as an empty sequence (e.g., an empty list) [1]. MultiIndex objects require valid, non-empty level sequences [1]. 2. Internal Accessor Registration: As cuDF frequently refactors its accessor architecture (e.g., moving accessors from cudf/core/column to dedicated namespaces) [2], issues can arise when a property accessor (like .list or .str) is invoked on an empty column that does not properly reflect the expected metadata/type structure required for that accessor [3][2]. 3. Data Invariants: cuDF strictly enforces Arrow-compatible invariants for its column formats [4][5]. Operations on empty list columns that do not satisfy these internal memory layout requirements—such as having an offset array size that does not match the expected row count—often trigger validation errors during read or conversion operations [4][5]. To troubleshoot this: - Verify that your input data is not an empty list or an empty DataFrame if you are attempting to construct a MultiIndex [1]. - Check for pandas compatibility mode settings (e.g., cudf.set_option("mode.pandas_compatible", True)), as this sometimes alters how empty indexes are handled [6]. - If you are using a specific version of cuDF, ensure your code is not relying on deprecated internal APIs, as recent updates have moved accessor implementations and removed internal cached accessors [7][2]. - If you can reproduce this with a minimal example, it is recommended to open an issue on the RAPIDS cuDF GitHub repository, as these errors are often tied to specific internal validation logic in pylibcudf or the Python-side wrappers [4][6].

Citations:

• 1: https://github.com/rapidsai/cudf/blob/release/26.02/python/cudf/cudf/core/multiindex.py
• 2: GitHub pull request 19134 in rapidsai/cudf (link omitted to avoid creating a cross-reference)
• 3: https://github.com/rapidsai/cudf/blob/branch-25.06/python/cudf/cudf/core/column/lists.py
• 4: GitHub issue 16164 in rapidsai/cudf (link omitted to avoid creating a cross-reference)
• 5: GitHub pull request 10935 in rapidsai/cudf (link omitted to avoid creating a cross-reference)
• 6: GitHub issue 17747 in rapidsai/cudf (link omitted to avoid creating a cross-reference)
• 7: GitHub pull request 22056 in rapidsai/cudf (link omitted to avoid creating a cross-reference)

🏁 Script executed:

#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import importlib.util
print("pandas_spec", importlib.util.find_spec("pandas"))
PY

Repository: NVIDIA/cudf

Length of output: 166

🌐 Web query:

pandas Index str split expand=True empty Index partition extract behavior official documentation source

💡 Result:

In pandas, the Index.str.split method with expand=True is designed to transform string elements in an Index into a MultiIndex by splitting them into separate components [1][2]. Behavioral and technical considerations include: 1. Return Type: According to the official documentation, when expand=True is used on an Index, the method returns a MultiIndex [1][2]. If expand=False (the default), it returns an Index containing lists of strings [1][2]. 2. Dimensionality and Splitting: The number of levels in the resulting MultiIndex depends on the number of splits performed. If a string cannot be split (e.g., the delimiter is not found), the behavior may vary depending on the pandas version; historically, some users have encountered unexpected behavior where the return type is not a MultiIndex in cases where no splits occur [3]. 3. Empty/Missing Data: - Empty Strings: Splitting an empty string typically results in an empty sequence or is handled gracefully in modern versions, though early versions (pre-0.22) faced exceptions [4]. - NaN Handling: Index.str.split with expand=True is not always fully "NaN-safe" across all historical versions and edge cases, sometimes triggering TypeErrors if the index contains non-string types like floats [5]. 4. Practical Limitations: When working with expand=True, the number of resulting columns (or levels) is dynamic, based on the input data. If you are assigning these results to a DataFrame or performing further operations, ensure your code accounts for cases where the split might yield fewer columns than expected (e.g., when data is missing or strings are inconsistent), as this can cause mismatches in dimensions [6]. For robust workflows, it is often recommended to verify the output shape or structure if your downstream logic depends on a fixed number of levels or columns, especially when dealing with heterogeneous or potentially incomplete string data [6].

Citations:

• 1: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.split.html
• 2: https://pandas.pydata.org/pandas-docs/version/3.0/reference/api/pandas.Series.str.split.html
• 3: GitHub issue 49867 in pandas-dev/pandas (link omitted to avoid creating a cross-reference)
• 4: GitHub issue 20002 in pandas-dev/pandas (link omitted to avoid creating a cross-reference)
• 5: GitHub issue 23677 in pandas-dev/pandas (link omitted to avoid creating a cross-reference)
• 6: https://stackoverflow.com/questions/57652702/how-to-make-pandas-series-str-split-work-with-empty-dataset

🌐 Web query:

site:pandas.pydata.org/docs/reference/api pandas Index.str.partition empty Index MultiIndex

💡 Result:

In pandas, when using Index.str.partition on an empty Index, the behavior depends on the parameters provided, particularly the default expand=True behavior. When you call str.partition on an empty Index, it returns an empty MultiIndex [1][2]. By design, this method splits strings into three parts (before, separator, after) and, by default, expands these into a MultiIndex to hold the resulting structure [1][2]. If the input Index is empty, the result retains this structure but contains no elements [3]. Key behaviors: 1. Default (expand=True): Calling idx.str.partition on an empty Index results in an empty MultiIndex [1][2]. 2. Alternative (expand=False): If you specify expand=False, the method returns an empty Index containing tuples rather than a MultiIndex [1][2]. An Index is considered empty if it contains no elements [3]. Because the str accessor methods are vectorized to handle the contents of the Index, they generally propagate the empty state of the underlying data while preserving the expected return type (e.g., a MultiIndex for expanded output) [1][2].

Citations:

• 1: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.partition.html
• 2: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.rpartition.html
• 3: http://pandas.pydata.org/docs/reference/api/pandas.Index.empty.html

🌐 Web query:

site:github.com/pandas-dev/pandas "Index([], dtype" "str.partition" MultiIndex

💡 Result:

The string 'Index([], dtype=str)' appearing alongside a MultiIndex in pandas typically indicates that a string operation (accessed via the.str accessor) has been performed on a pandas Index object that contains string data, resulting in a structured, multi-level output [1][2]. Specifically, when using methods like str.partition or str.split on a pandas Index with expand=True (the default), pandas returns a MultiIndex rather than a standard Index to represent the split components of the strings [1][3]. For example, performing a partition on an index: idx = pd.Index(["X 123", "Y 999"]) # This creates a MultiIndex idx.str.partition If you encounter an Index object with dtype=str or dtype=object appearing in your output when working with these methods, it is often because expand=False was specified, which forces the operation to return an Index containing tuples or objects instead of expanding into a MultiIndex [1][2][3]. Key points regarding this behavior: 1. MultiIndex output: Operations like.str.partition or.str.split naturally return a MultiIndex when expanding the results [1]. 2. expand parameter: You can control this behavior using the expand parameter. Setting expand=False will return an Index of objects (containing the resulting tuples) instead of a MultiIndex [1][3]. 3. Method functionality: These operations are designed to handle string data within Index objects, provided the underlying data consists of strings [4][5]. Note that these accessors generally do not work on MultiIndex objects themselves [4][5].

Citations:

• 1: https://github.com/pandas-dev/pandas/blob/main/pandas/core/strings/accessor.py
• 2: https://github.com/pandas-dev/pandas/blob/v3.0.4/pandas/core/strings/accessor.py
• 3: https://github.com/pandas-dev/pandas/blob/v1.3.0rc0/pandas/core/strings/accessor.py
• 4: https://github.com/pandas-dev/pandas/blob/1.0.x/pandas/core/strings.py
• 5: https://github.com/pandas-dev/pandas/blob/v2.0.2/pandas/core/strings/accessor.py